Clinical Application to Improve the "Depth Perception Problem" by Combining Augmented Reality and a 3D Printing Model.

In our experience with intraoperative evaluation and educational application of augmented reality technology, an illusion of depth has been a major problem. To improve this depth perception problem, we conducted two experiments combining various three-dimensional models and holograms and the observation angles using an augmented reality device. Methods: In experiment 1, when observing holograms projected on the surface layer of the model (bone model) or holograms projected on a layer deeper than the model (body surface model), the observer's first impression regarding which model made it easier to understand positional relationships was investigated. In experiment 2, to achieve a more quantitative evaluation, the observer was asked to measure the distance between two specific points on the surface and deep layers from two angles in each of the above combinations. Statistical analysis was performed on the measurement error for this distance. Results: In experiment 1, the three-dimensional positional relationships were easier to understand in the bone than in the body surface model. In experiment 2, there was not much difference in the measurement error under either condition, which was not large enough to cause a misunderstanding of the depth relationship between the surface and deep layers. Conclusions: Any combination can be used for preoperative examinations and anatomical study purposes. In particular, projecting holograms on a deep model or observing positional relationships from not only the operator's viewpoint, but also multiple other angles is more desirable because it reduces confusion caused by the depth perception problem and improves understanding of anatomy.

Application of augmented reality (AR) technology to locate the cutaneous perforator of anterolateral thigh perforator flap: A case report.

Multi-detector row computed tomography (MDCT) makes it possible to visualize the peripheral perforators of the anterolateral thigh (ALT) flap. However, to transfer the preoperative MDCT angiography data to the operative field requires effective techniques. In this report, we describe an application of augmented reality (AR) technology to harvest the anterolateral thigh flap. A 36-year-old female presented with a T2N2 squamous cell carcinoma of the lateral tongue. The patient underwent hemiglossectomy and microsurgical reconstruction using the left ALT flap. Three dimensional (3D) images the vascular image, vascular with muscles and vascular with outline of the thigh ware prepared. Then these images were exposed to an AR device. The location of the perforator was determined using the 3D vascular image on AR. The intraoperative location of the cutaneous perforator corresponded with the predicted location which was confirmed using the AR technique. A 6 × 15 cm left ALT flap was transferred to the defect. Microsurgical anastomosis was performed on the left superior thyroid artery and the internal jugular vein. There were no complications during the postoperative course. At the 6-month follow-up, the patient showed no evidence of flap and donor site complications. Our experience suggests that AR technology may effectively support the transfer of MDCT angiography images onto surgical sites.

Clinical Applications of Meshed Multilayered Anatomical Models by Low-Cost Three-Dimensional Printer.

SUMMARY: In recent years, even low-cost fused deposition modeling-type three-dimensional printers can be used to create a three-dimensional model with few errors. The authors devised a method to create a three-dimensional multilayered anatomical model at a lower cost and more easily than with established methods, by using a meshlike structure as the surface layer. Fused deposition modeling-type three-dimensional printers were used, with opaque polylactide filament for material. Using the three-dimensional data-editing software Blender (Blender Foundation, www.blender.org) and Instant Meshes (Jakob et al., https://igl.ethz.ch/projects/instant-meshes/) together, the body surface data were converted into a meshlike structure while retaining its overall shape. The meshed data were printed together with other data (nonmeshed) or printed separately. In each case, the multilayer model in which the layer of the body surface was meshed could be output without any trouble. It was possible to grasp the positional relationship between the body surface and the deep target, and it was clinically useful. The total work time for preparation and processing of three-dimensional data ranged from 1 hour to several hours, depending on the case, but the work time required for converting into a meshlike shape was about 10 minutes in all cases. The filament cost was $2 to $8. In conclusion, the authors devised a method to create a three-dimensional multilayered anatomical model to easily visualize positional relationships within the structure by converting the surface layer into a meshlike structure. This method is easy to adopt, regardless of the available facilities and economic environment, and has broad applications.

Intraoperative 3-dimensional Projection of Blood Vessels on Body Surface Using an Augmented Reality System.

Preoperative understanding of the running pattern of blood vessels is an important factor to approach surgical fields safely. In 2 cases where the vascular abnormalities were estimated, we projected the blood vessels onto the surgical field using an augmented reality device HoloLens. A splint was made to allow the patient to be fixed while undergoing computed tomographic angiography. Three-dimensional (3D) data on the blood vessels, skin surfaces, bones, and the 3 chosen points for alignment were segmented and then projected onto the body surfaces as holograms using the HoloLens. Two types of projection for holograms were used: projection type 1-where the body contours were projected as a line, and projection type 2-where the body surface was projected as meshed skin type. By projecting projection type 2 rather than projection type 1, we gained a better understanding of the 3D anatomic findings and deformation characteristics, including the anatomic blood vessel variation and positional relationships between the organs and body surfaces. To some extent, we could make sure that the depth perception can be obtained by recognizing the bone, vessels, or tumor inside the meshed skin surface. Our new method allows the 3D visualization of blood vessels from the body surface, and helps understand the 3D anatomic variation of the blood vessels to be applied as long as the blood vessels can be visualized.

Augmented Reality Technology for the Positioning of the Auricle in the Treatment of Microtia.

BACKGROUND: The positioning of the auricle is a key factor in successful ear reconstruction. However, the position of the ear is usually determined by transferring the auricle image of the nonaffected side to the affected side using a transparent film. Augmented reality (AR) is becoming useful in the surgical field allowing computer-generated images to be superimposed on patients. In this report, we would like to introduce an application of AR technology in ear reconstruction. METHODS: AR technology was used to determine the position of the reconstructed ear of a 10-year-old male with right microtia. Preoperative 3-dimensional photographs of the nonaffected side were taken using VECTRAH1. Then, the image was horizontally inverted and superimposed on the three-dimensional image of the affected side with reference to the anatomical landmarks of the patient's face. These images were projected onto the patient in the operation room using Microsoft's HoloLens. The design and positioning of the auricle was done with reference to the AR image. To confirm the accuracy of the AR technique, we compared it to the original transparent film technique. After the insertion of the cartilage framework into the skin pocket, the position and shape of the reconstructed ear was confirmed using the AR technology. RESULTS: The positioning of the reconstructed ear was successfully performed. The deviation between the 2 designated positions using the AR and the transparent film was within 2 mm. CONCLUSION: The AR technology is a promising option in the surgical treatment of microtia.

Three-dimensional Camera Imaging in Postoperative Evaluation of Distraction Osteogenesis.

Distraction osteogenesis needs to be regularly assessed in some way to monitor the degree of advancement. X-ray is used for the general evaluation of osteotomy. However, radiation exposure should be avoided. The purpose of this study is to evaluate 3-dimensional (3D) camera imaging for postoperative evaluation. Three patients who underwent Le Fort I or III advancement osteotomy using rigid external distraction and internal distraction were observed in this study. The degrees of the distractions were evaluated using VECTRA H1 3D imaging in addition to computed tomographic (CT) scans. In the VECTRA 3D imaging, the tilt and size of the faces were corrected using the dedicated software for imaging. The preoperative and postoperative images were superimposed, and the distances of motion between the landmarks were measured. In CT scans, the bone distances between osteotomy points of the pterygomaxillary junction were analyzed. As the VECTRA 3D imaging can be compared by overlaying previous photographs, it served as a good tool to evaluate the distractions. However, both the soft-tissue movement measured by VECTRA and CT bony measurements did not match the total amount of movement for the internal distraction devices. The bony advancements were less than the amount of distraction. The soft tissues shrank after the distraction was completed in all cases. Three-dimensional camera imaging is considered to be a useful tool for the evaluation of distraction osteogenesis.

Effective Application of Mixed Reality Device HoloLens: Simple Manual Alignment of Surgical Field and Holograms.

The technology used to add information to a real visual field is defined as augmented reality technology. Augmented reality technology that can interactively manipulate displayed information is called mixed reality technology. HoloLens from Microsoft, which is a head-mounted mixed reality device released in 2016, can display a precise three-dimensional model stably on the real visual field as hologram. If it is possible to accurately superimpose the position/direction of the hologram in the surgical field, surgical navigation-like use can be expected. However, in HoloLens, there was no such function. The authors devised a method that can align the surgical field and holograms precisely within a short time using a simple manual operation. The mechanism is to match the three points on the hologram to the corresponding marking points of the body surface. By making it possible to arbitrarily select any of the three points as a pivot/axis of the rotational movement of the hologram, alignment by manual operation becomes very easy. The alignment between the surgical field and the hologram was good and thus contributed to intraoperative objective judgment. By using the method of this study, the clinical usefulness of the mixed reality device HoloLens will be expanded.

Telementoring Demonstration in Craniofacial Surgery With HoloLens, Skype, and Three-Layer Facial Models.

BACKGROUND: Telementoring is the technology for providing surgical instruction from a remote place via a network. To demonstrate the use of telementoring in craniofacial surgery, Skype and a mixed reality device HoloLens were adopted, and 3-layer facial models had been developed. METHODS: A resident in hospital A used the model surgery under remote guidance by a mentor surgeon in hospital B 4 times on different dates. The straight-line between hospitals A and B is 250 km. The mentor gave the resident guidance via Skype and HoloLens, communicating by voice, and video of the surgical field, and providing reference data. RESULTS: There was no delay in voice communication and a delay of <0.5 seconds in the video. The resident was able to confirm the main landmarks of the surgical field and to grasp the situation without problems. The mentor could send appropriate instructions by voice, could point out a specific part by telestration function, and could draw lines on the 2-dimentional images pasted on the operator's field of vision. DISCUSSION: With the use of HoloLens, Skype, and the 3-layer models, it was possible to demonstrate telementoring. The risk of personal information leakage due to data interception seems to be very low because its data communication is encrypted with advanced encryption standard. CONCLUSION: This telementoring system has various advantages and many improvable aspects in the field of craniofacial surgery.

Three-Dimensional Computer-Assisted Two-Layer Elastic Models of the Face.

To make three-dimensional computer-assisted elastic models for the face, we decided on five requirements: (1) an elastic texture like skin and subcutaneous tissue; (2) the ability to take pen marking for incisions; (3) the ability to be cut with a surgical knife; (4) the ability to keep stitches in place for a long time; and (5) a layered structure. After testing many elastic solvents, we have made realistic three-dimensional computer-assisted two-layer elastic models of the face and cleft lip from the computed tomographic and magnetic resonance imaging stereolithographic data. The surface layer is made of polyurethane and the inner layer is silicone. Using this elastic model, we taught residents and young doctors how to make several typical local flaps and to perform cheiloplasty. They could experience realistic simulated surgery and understand three-dimensional movement of the flaps.

Intraoperative Evaluation of Body Surface Improvement by an Augmented Reality System That a Clinician Can Modify.

BACKGROUND: Augmented reality (AR) technology that can combine computer-generated images with a real scene has been reported in the medical field recently. We devised the AR system for evaluation of improvements of the body surface, which is important for plastic surgery. METHODS: We constructed an AR system that is easy to modify by combining existing devices and free software. We superimposed the 3-dimensional images of the body surface and the bone (obtained from VECTRA H1 and CT) onto the actual surgical field by Moverio BT-200 smart glasses and evaluated improvements of the body surface in 8 cases. RESULTS: In all cases, the 3D image was successfully projected on the surgical field. Improvement of the display method of the 3D image made it easier to distinguish the different shapes in the 3D image and surgical field, making comparison easier. In a patient with fibrous dysplasia, the symmetrized body surface image was useful for confirming improvement of the real body surface. In a patient with complex facial fracture, the simulated bone image was useful as a reference for reduction. In a patient with an osteoma of the forehead, simultaneously displayed images of the body surface and the bone made it easier to understand these positional relationships. CONCLUSIONS: This study confirmed that AR technology is helpful for evaluation of the body surface in several clinical applications. Our findings are not only useful for body surface evaluation but also for effective utilization of AR technology in the field of plastic surgery.